Skip to content

Release v2.0.0 with AWS cloud support.#7

Open
NinjaRocks wants to merge 12 commits intomasterfrom
release/v2.0.0-aws
Open

Release v2.0.0 with AWS cloud support.#7
NinjaRocks wants to merge 12 commits intomasterfrom
release/v2.0.0-aws

Conversation

@NinjaRocks
Copy link
Member

SourceFlow.Net v2.0.0 - Changelog

Release Date: TBC
Status: In Development

Note: This release includes AWS cloud integration support. Azure cloud integration will be available in a future release.

🎉 Major Changes

Cloud Core Consolidation

The SourceFlow.Cloud.Core project has been consolidated into the main SourceFlow package. This architectural change simplifies the dependency structure and reduces the number of separate packages required for cloud integration.

Benefits:

  • ✅ Simplified package management (one less NuGet package)
  • ✅ Reduced build complexity
  • ✅ Improved discoverability (cloud functionality is part of core)
  • ✅ Better performance (eliminates one layer of assembly loading)
  • ✅ Easier testing (no intermediate package dependencies)

✨ New Features

Integrated Cloud Functionality

The following components are now part of the core SourceFlow package:

Configuration

  • BusConfiguration - Fluent API for routing configuration
  • IBusBootstrapConfiguration - Bootstrapper integration
  • ICommandRoutingConfiguration - Command routing abstraction
  • IEventRoutingConfiguration - Event routing abstraction
  • IIdempotencyService - Duplicate message detection
  • InMemoryIdempotencyService - Default implementation
  • IdempotencyConfigurationBuilder - Fluent API for idempotency configuration

Resilience

  • ICircuitBreaker - Circuit breaker pattern interface
  • CircuitBreaker - Implementation with state management
  • CircuitBreakerOptions - Configuration options
  • CircuitBreakerOpenException - Exception for open circuits
  • CircuitBreakerStateChangedEventArgs - State transition events

Security

  • IMessageEncryption - Message encryption abstraction
  • SensitiveDataAttribute - Marks properties for encryption
  • SensitiveDataMasker - Automatic log masking
  • EncryptionOptions - Encryption configuration

Dead Letter Processing

  • IDeadLetterProcessor - Failed message handling
  • IDeadLetterStore - Failed message persistence
  • DeadLetterRecord - Failed message model
  • InMemoryDeadLetterStore - Default implementation

Observability

  • CloudActivitySource - OpenTelemetry activity source
  • CloudMetrics - Standard cloud metrics
  • CloudTelemetry - Centralized telemetry

Serialization

  • PolymorphicJsonConverter - Handles inheritance hierarchies

Idempotency Configuration Builder

New fluent API for configuring idempotency services:

// Entity Framework-based (multi-instance)
var idempotencyBuilder = new IdempotencyConfigurationBuilder()
    .UseEFIdempotency(connectionString, cleanupIntervalMinutes: 60);

// In-memory (single-instance)
var idempotencyBuilder = new IdempotencyConfigurationBuilder()
    .UseInMemory();

// Custom implementation
var idempotencyBuilder = new IdempotencyConfigurationBuilder()
    .UseCustom<MyCustomIdempotencyService>();

// Apply configuration
idempotencyBuilder.Build(services);

Builder Methods:

  • UseEFIdempotency(connectionString, cleanupIntervalMinutes) - Entity Framework-based (requires SourceFlow.Stores.EntityFramework package)
  • UseInMemory() - In-memory implementation
  • UseCustom<TImplementation>() - Custom implementation by type
  • UseCustom(factory) - Custom implementation with factory function

Enhanced AWS Integration

AWS cloud extension now supports explicit idempotency configuration:

services.UseSourceFlowAws(
    options => { options.Region = RegionEndpoint.USEast1; },
    bus => bus.Send.Command<CreateOrderCommand>(q => q.Queue("orders.fifo")),
    configureIdempotency: services =>
    {
        services.AddSourceFlowIdempotency(connectionString);
    });

📚 Documentation Updates

New Documentation

Updated Documentation

🐛 Bug Fixes

  • None (this is a major architectural release)

🔧 Internal Changes

Project Structure

  • Consolidated src/SourceFlow.Cloud.Core/ into src/SourceFlow/Cloud/
  • Simplified dependency graph for cloud extensions
  • Reduced NuGet package count

Build System

  • Updated project references to remove Cloud.Core dependency
  • Simplified build pipeline
  • Reduced compilation time

📦 Package Dependencies

SourceFlow v2.0.0

  • No new dependencies added
  • Cloud functionality now integrated

SourceFlow.Cloud.AWS v2.0.0

  • Depends on: SourceFlow >= 2.0.0
  • Removed: SourceFlow.Cloud.Core dependency

🚀 Upgrade Path

For AWS Extension Users

If you're using the AWS cloud extension, no code changes are required. The consolidation is transparent to consumers of the cloud package.

📝 Notes

  • This is a major version release due to breaking namespace changes
  • The consolidation improves the overall architecture and developer experience
  • All functionality from Cloud.Core is preserved in the main SourceFlow package
  • AWS cloud extension remains a separate package with simplified dependencies
  • Azure cloud integration will be available in a future release

🔗 Related Documentation


Version: 2.0.0
Date: TBC
Status: In Development

var masker = new SensitiveDataMasker();
var masked = masker.Mask(testData);

_logger.LogInformation("Masked data: {MaskedData}", masked);

Check warning

Code scanning / CodeQL

Exposure of private information Medium test

Private data returned by
call to method MaskCreditCard
is written to an external location.
Private data returned by
call to method MaskEmail
is written to an external location.

Copilot Autofix

AI about 1 hour ago

In general, the safest way to address this class of issue is to avoid writing sensitive or potentially sensitive data to external locations (logs, files, etc.), even in masked form, unless strictly necessary. When logging is needed, it should be limited to metadata (for example, whether masking succeeded) rather than the actual masked content.

For this specific case, we do not need to log the masked string at all to validate functionality: the subsequent Assert.* statements already check that the masked string does not contain original secrets and does contain masking tokens. The minimal, non‑functional change is therefore to remove or neutralize the _logger.LogInformation("Masked data: {MaskedData}", masked); line. If some logging is still desired, we can log a generic message that masking completed, without including the masked variable. No changes are required in SensitiveDataMasker itself; it is only referenced by the test.

Concretely:

  • In tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs, in the SensitiveDataMasking_WithCreditCardAttribute_ShouldMaskInLogs test, replace the line that logs masked with a log line that does not embed the sensitive data (or remove it entirely). Everything else in this file and in SensitiveDataMasker.cs can remain unchanged for this alert.
Suggested changeset 1
tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs b/tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs
--- a/tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs
+++ b/tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs
@@ -78,7 +78,8 @@
         var masker = new SensitiveDataMasker();
         var masked = masker.Mask(testData);
         
-        _logger.LogInformation("Masked data: {MaskedData}", masked);
+        // Do not log masked data content to avoid exposing sensitive information, even in masked form
+        _logger.LogInformation("Sensitive data masking completed for test payload.");
         
         // Verify masked output doesn't contain full sensitive values
         Assert.DoesNotContain("4532-1234-5678-9010", masked);
EOF
@@ -78,7 +78,8 @@
var masker = new SensitiveDataMasker();
var masked = masker.Mask(testData);

_logger.LogInformation("Masked data: {MaskedData}", masked);
// Do not log masked data content to avoid exposing sensitive information, even in masked form
_logger.LogInformation("Sensitive data masking completed for test payload.");

// Verify masked output doesn't contain full sensitive values
Assert.DoesNotContain("4532-1234-5678-9010", masked);
Copilot is powered by AI and may make mistakes. Always verify output.
Assert.DoesNotContain("MyP@ssw0rd!", masked);
Assert.DoesNotContain("pk_live_abcdefghijklmnopqrstuvwxyz123456", masked);

_logger.LogInformation("Comprehensive masked data: {MaskedData}", masked);

Check warning

Code scanning / CodeQL

Exposure of private information Medium test

Private data returned by
call to method MaskCreditCard
is written to an external location.
Private data returned by
call to method MaskEmail
is written to an external location.

Copilot Autofix

AI about 1 hour ago

General approach: avoid writing potentially sensitive (or derived-from-sensitive) data to external sinks such as logs, especially in ways that might re-expose information if masking is incomplete. In this case, we can preserve the test’s behavior (assertions about masking) while omitting the actual masked payload from the log.

Best concrete fix: update the integration test so that it no longer logs the masked JSON string. Instead, log a generic success message that does not include the masked data as a structured argument. The assertions already validate that sensitive literals are not present, so logging the content is unnecessary.

Specific changes:

  • File: tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs
    • In SensitiveDataMasking_WithMultipleTypes_ShouldMaskAllCorrectly, replace the line:
      • _logger.LogInformation("Comprehensive masked data: {MaskedData}", masked);
        with something like:
      • _logger.LogInformation("Comprehensive sensitive data masking test completed successfully.");
        This keeps the log useful without sending the masked JSON to the logger sink.

No changes are needed in SensitiveDataMasker itself to resolve this CodeQL alert, and no new imports or helpers are required.


Suggested changeset 1
tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs b/tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs
--- a/tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs
+++ b/tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs
@@ -122,7 +122,7 @@
         Assert.DoesNotContain("MyP@ssw0rd!", masked);
         Assert.DoesNotContain("pk_live_abcdefghijklmnopqrstuvwxyz123456", masked);
         
-        _logger.LogInformation("Comprehensive masked data: {MaskedData}", masked);
+        _logger.LogInformation("Comprehensive sensitive data masking test completed successfully.");
     }
     
     #endregion
EOF
@@ -122,7 +122,7 @@
Assert.DoesNotContain("MyP@ssw0rd!", masked);
Assert.DoesNotContain("pk_live_abcdefghijklmnopqrstuvwxyz123456", masked);

_logger.LogInformation("Comprehensive masked data: {MaskedData}", masked);
_logger.LogInformation("Comprehensive sensitive data masking test completed successfully.");
}

#endregion
Copilot is powered by AI and may make mistakes. Always verify output.
Major release with AWS cloud integration, CI/CD enhancements, and comprehensive testing improvements.

## AWS Cloud Integration
- Add AWS SQS/SNS integration for distributed command and event processing
- Implement LocalStack support for local AWS service emulation
- Add comprehensive AWS integration tests with property-based testing
- Fix LocalStack connectivity and authentication in CI environments
- Add external LocalStack detection to prevent container conflicts

## CI/CD Improvements
- Configure LocalStack as GitHub Actions service for integration tests
- Add NuGet cache clearing to prevent stale package metadata issues
- Exclude integration and security tests from CI (run unit tests only)
- Add comprehensive GitHub Actions setup documentation
- Fix GitVersion configuration for release branches
- Update workflows with paths-ignore for documentation changes

## Testing Enhancements
- Add LocalStack timeout and connectivity diagnostics
- Implement property-based tests for AWS service equivalence
- Add dead letter queue processing tests
- Fix SQS queue attribute names and DLQ test timing
- Add CI-optimized LocalStack configuration with extended timeouts

## Documentation
- Add GitHub Actions setup guide with troubleshooting
- Update cloud integration testing documentation
- Add AWS cloud architecture documentation
- Update README with new logo images

## Bug Fixes
- Fix .NET Standard 2.1 compatibility with GlobalUsings.cs
- Fix AWS client endpoint configuration for LocalStack
- Fix IAM enforcement in LocalStack service container
- Fix compilation errors in AwsTestConfiguration

## Breaking Changes
- Cloud.Core functionality consolidated into main SourceFlow package (v2.0.0)
- Namespace changes: SourceFlow.Cloud.Core.* → SourceFlow.Cloud.*

Related specs:
- .kiro/specs/v2-0-0-release-preparation/
- .kiro/specs/github-actions-localstack-timeout-fix/
- .kiro/specs/github-actions-ci-configuration-fix/
@NinjaRocks NinjaRocks force-pushed the release/v2.0.0-aws branch from abb7ad7 to 3db7487 Compare March 7, 2026 22:13
- Release branches now generate pre-release packages with -beta suffix
- Example: 2.0.0-beta.1 instead of 2.0.0
- Prevents accidental stable release publication from release branches
- Final stable releases still triggered by 'release-packages' tag
@NinjaRocks NinjaRocks requested a review from Nshai March 15, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant